Method and apparatus for automatically identifying and annotating auditory signals from one or more parties

ABSTRACT

A telecommunications system, that after a communication is established by a first electronic communication device and a second electronic communication device, receives digital auditory data corresponding to an ongoing conversation between a first person using the first electronic communication device and a second person using the second electronic communication device, uses that digital auditory data to enable transformation of an information knowledge model, and utilizes the information knowledge model to suggest information for the ongoing conversation.

PRIORITY CLAIM

This continuation application claims priority to and the benefit of U.S.Non-Provisional patent application Ser. No. 16/780,201, filed Feb. 3,2020, which is a continuation application claiming priority to and thebenefit of U.S. Non-Provisional patent application Ser. No. 16/280,266,filed Feb. 20, 2019, now U.S. Pat. No. 10,582,350, granted on Mar. 3,2020, which is a continuation application claiming priority to and thebenefit of U.S. Non-Provisional patent application Ser. No. 15/927,198,filed Mar. 21, 2018, now U.S. Pat. No. 10,244,368, granted on Mar. 26,2019, which is a continuation application claiming priority to and thebenefit of U.S. Non-Provisional patent application Ser. No. 14/208,278,filed Mar. 13, 2014, now U.S. Pat. No. 10,070,283, granted on Sep. 4,2018, which claims priority to and the benefit of U.S. ProvisionalPatent Application No. 61/789,945, filed Mar. 15, 2013, the entirecontents of all of which are incorporated herein by reference.

TECHNICAL FIELD

The present disclosure relates generally to telecommunications overwireless networks.

BACKGROUND

Users of smart phones and other mobile devices are often given importantinformation during a conversation. Circumstances often make taking notesdifficult and the user is will lose the information unless it can beremembered

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an example portable multifunctional device;and

FIG. 2 is a block diagram illustrating and example server or clientcomputer workstation.

DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS Overview

Embodiments of the invention address the need to identify and annotateauditory data streams with structural information. For example, in atelephone conversation between two parties using cellular smart phones,the phone application would process, either locally or remotely or anycombination, the auditory data stream to identify the mention ofspecific entities, e.g. telephone numbers, email addresses, dates andtimes, locations, etc. These identified entities are displayed to theparties who may indicate actions to perform with or on that entity,including disregarding due to non-relevance, inaccuracy, imprecision, orother attribute that may be used to train the underlying system oncontexts in which those entities are relevant and what additionalaction, e.g. adding to contacts, calendar, or other applications on themobile device.

Description

Reference will now be made in detail to various embodiments of theinvention. Examples of these embodiments are illustrated in theaccompanying drawings. While the invention will be described inconjunction with these embodiments, it will be understood that it is notintended to limit the invention to any embodiment. On the contrary, itis intended to cover alternatives, modifications, and equivalents as maybe included within the spirit and scope of the invention as defined bythe appended claims. In the following description, numerous specificdetails are set forth in order to provide a thorough understanding ofthe various embodiments. However, various embodiments may be practicedwithout some or all of these specific details. In other instances, wellknown process operations have not been described in detail in order notto unnecessarily obscure the present invention. Further, each appearanceof the phrase an “example embodiment” at various places in thespecification does not necessarily refer to the same example embodiment.

In an example embodiment the cellular phone application (CPA) utilizesthe intervening telecommunications infrastructure to route digitalauditory data in real-time to a cloud-based phone recognition and entityidentification, annotation and relevance processing which in turn sendsselected relevant annotated entities to said CPA for use in interactingwith user. In this example embodiment the CPA displays such annotatedentities as they are received, which may or may not synchronize inreal-time with said auditory stream, either as a result from automatedprocessing (e.g. “A new calendar entry has been created for XX on YY atZZ”) or for subsequent user interaction prior to additional processing,e.g. in a conversation between two users, Bob and Paul, the CPA makesuse of Bob's local address book to know the name of the other party,Paul For example, a possible conversation, and the corresponding CPAresponse would be:

Bob: What number should I call?

Paul: I would call 800-555-1212 and then enter conference code 12345followed by the hash sign.

Bob: Thanks let me find a pen.

CPA: “Paul mentioned a phone number (800-555-1212); would you like me torecord that number?”

1. Ignore, not relevant.

2. No, thank you.

3. Yes, as a new number for Paul.

4. Yes, as number to call.

Bob: <indication of option 4>

Bob: Paul, don't worry, I got it.

CPA: “Paul mentioned what sounded like a meeting code after thetelephone number you asked to call, 12345#, would you like me to addthat to the telephone number?”

1. Ignore, not relevant

2. No, thank you.

3. Yes, as a meeting code.

4. Yes, as another number.

Bob: <indication of option 3>

Bob: Paul, thanks; I got it. I will talk to you in a couple minutes.

Paul: Bob, talk to you soon.

CPA: Would you like to call telephone number 800-555-1212 meeting code12345# now or later?

1. Ignore, not relevant

2. No, please remind me later.

3. Yes, please call now.

4. Yes, let's schedule it.

Bob: <indication of option 3>

In an alternative embodiment (AE), the CPA audio stream is monitored bya text-to-speech application, running in the background on the user'shandset, that converts the audio signal to text, to create a real-timetextual transcript of the conversation upon the AE can act to performknowledge management functions. This transcript is presented to the userin a visual display similar in layout to a txt chat application such asiMessage, wherein each interchange of the conversation is presented tothe viewer in a shaded message ‘bubble’ with the user's communicationsaligned along the right edge of the application window, and the remoteparticipant's communications aligned along the left.

Pauses longer than an adjustable time threshold cause new messagebubbles to be created. As the conversation proceeds, the message bubblewindow continuously scrolls the conversation transcript upwards.

If the user sees information within a message bubble that the user isdesirous to have knowledge management functions performed upon, the usercan tap on that message bubble to cause an option-selection dialog to bepresented whereby the user can choose the desired function to beperformed.

The choices presented are dependent upon the content and the context ofthe text within the selected message bubble. Some choices may be, forexample, to have the AE create a new appointment in the user's calendarapp based upon the subject, date and time mentioned in the messagebubble, or to check reservation availability at a restaurant based uponthe mention of the restaurant's name and a date and time in the messagebubble.

In another alternative embodiment (AE2), the system can be used to buildand complete an information model via annotated audio stream processing.Using a speech-to-text converter in conjunction with an entityextraction and identification text processing system, a stream of wordswith associated annotated objects entities is emitted by capturing thespeech of parties engaged in knowledge engineering an information modelor completing an instance of that model.

In both instances, either a meta-information model or a specificinformation model, the speech-to-text-to-entities process yields objectsthat are defined in the model. For example, an interviewer (A) andinterviewee (B) engaged in completing a information model correspondingto a police report. In this example, the interviewer (A) would beprompted with lines of inquiry about high priority entities andrelationships in the model, e.g. victim, alleged perpetrator, arrestingofficer, et. al. as well as the relationships amongst those entities(e.g. <arresting officer> may be different from <first officer onscene>). Additionally, as entities are spoken and identified, includingpronouns and other alternative indicators of said entities, theidentification as well as the disambiguation of entities (“the officer”refers to which previously mentioned <officer> or spoken “he” may referto victim, perpetrator, officer, et. al.

The information model guides the interview process to continuallyidentify high priority lines of inquiry, disambiguation of entities,relationships, attributes and values, etc. as the interview proceeds;even with relatively long delays in processing the speech to relevantand priorities entities (e.g. several minutes), the dynamic nature ofthe process allows for prior subjects to be revisited when necessary.

Yet another embodiment involves the incorporation into the system of theAE a fact-checking module. The fact-checking module is a systemcomponent that, provides for an additional message-bubble-selectionchoice: an option to have the system automatically parse and analyze thecontent of the message bubble so that the system can discerndistinguishable facts from the message content, and proceed toautomatically search online reference materials for relevantinformation, and to present that information to the user. This will beparticularly useful, for example, to a newspaper reporter doing a phoneinterview, so that the reporter can check on the factual basis ofassertions being made by the interviewee, even while the interview isunderway.

FIG. 1 shows an enlarged view of the portable (mobile) multifunctionaldevice 104. For example, if the portable multifunctional device 104 is acellular telephone, then the portable device circuitry includes acommunication link that may support protocols such as Global System forMobile communication (GSM), General Packet Radio Service (GPRS),Enhanced Data Rates for Global Evolution (EDGE), Universal MobileTelecommunications Service (UMTS), etc. The communication interface ofthe device 104 may also/alternately support Wireless Wide Area Network(WWAN), Wireless Local Area Network (WLAN), and/or Wireless PersonalArea Network (WPAN), etc.

Device 104 may further include a contactless element 164, typicallyimplemented in the form of a semiconductor chip 164(a) with anassociated wireless data transfer (e.g., data transmission) element164(b), such as an antenna. Contactless element 164 is associated with(e.g., embedded within) portable device 104 and data such as a coupon orcontrol instructions transmitted via cellular network may be applied tocontactless element 164 by means of contactless element interface 166.Contactless element interface 166 functions to permit the exchange ofdata and/or control instructions between the portable device circuitry162 (and hence the cellular network) and contactless element 164.

The contactless element may also include a Near Field Communication(NFC) module or other near field wireless reader module that allows theportable multifunctional device to communicate with a point of saleterminal (POS) at a merchant location by tapping the portablemultifunctional device to a reader.

Contactless element 164 is capable of transferring and receiving datausing a near field communications capability (or near fieldcommunications medium) typically in accordance with a standardizedprotocol or data transfer mechanism (identified as ISO 14443/NFC in thefigure). Near field communications capability is a short-rangecommunications capability, such as RFID, infra-red, or other datatransfer capability that can be used to exchange data between theportable device 104 and a local apparatus by tapping the portable deviceto the local apparatus, for example located at point-of-sale of amerchant or another location at which coupons are expected to beredeemed. Thus, portable device 104 is capable of communicating andtransferring data and/or control instructions via both cellular networkand near field communications capability.

Portable device 104 may also include a secure data space 168, which maybe used by the device to store operating parameters and/or other datautilized in operation of the device. The secure data space 168 may be inthe form of a chip that is separate and apart from the chip in thecontactless element 164, or alternatively, could be a section of memoryin the chip that forms part of the contactless element 164. Note thatthe chip in the contactless element 164 may include data storagecapability in the form of a memory that may be accessed via interface166 to permit the implementation of read, write, and erase functions,for example.

In accordance with still other embodiments, the portable device mayfurther include a processor 170 and computer readable storage medium 172for storing code and configured to direct the processor to performvarious tasks. For example, the computer readable storage medium maycomprise a magnetic disk drive or a flash memory chip. A smart phoneincludes an operating system such as Google Android or Apple iOSoperating system.

The computer readable storage medium may contain code that is configuredto cause a processor of the portable consumer device to receive andrecognize a message including a coupon and code that is delivered to theportable device. The computer readable storage medium may also includecode that is configured to decrypt an encrypted message including thecode that is received by the portable device.

In accordance with certain embodiments, the portable device 104 furtherincludes a Global Positioning System (GPS) element 174. GPS element 174is configured to allow determination of the location of the user at anytime. In particular, GPS element 174 relies upon signals from aplurality of orbiting satellites in order to allow the user's locationto be determined. Location information obtained from the GPS element 174may in turn be communicated through the antenna to allow monitoring ofthe user's position.

The GPS receiver determines a geographic location for the device bycalculating a distance between the device and at least three satellitesusing low-power radio signals received from the satellites using atechnique known as Trilateration, which is known in the art.

The portable multifunctional device includes an input interface 176 suchas, for example, a touch screen, keypad (which for present purposes willbe understood to include the other buttons, switches and keys referredto or may be implemented as soft keys on the display) for receiving userinput, a display component for displaying output information to the userand conventional receive/transmit circuitry. Other suitable inputinterfaces include a light pen, track ball, data glove, microphone, etc.The portable multifunctional device also includes an input/outputinterface that may include a keypad, a mouse, a screen, a touch screen,and/or any other type of interface that allows a user of the device tointeract with the device.

FIG. 2 is an illustration of basic subsystems in a client or servercomputer system workstation usable to implement cloud based computing.In FIG. 2 , subsystems are represented by blocks such as centralprocessor 180, non-transitory system memory 181 consisting of randomaccess memory (RAM) and/or read-only memory (ROM), display adapter 182,monitor 183, etc. The subsystems are interconnected via a system bus184. Additional subsystems such as a printer, keyboard, fixed disk andothers are shown. Peripherals and input/output (I/O) devices can beconnected to the computer system by, for example serial port 185. Forexample, serial port 185 can be used to connect the computer system to amodem for connection to a network or serial port 185 can be used tointerface with a mouse input device. The interconnection via system bus184 allows central processor 180 to communicate with each subsystem andto control the execution of instructions from system memory 181 or fixeddisk 186, and the exchange of information between subsystems.

Other arrangements of subsystems and interconnections are possible.

Some example embodiments are implemented as program code embodied in anon-transitory computer readable storage medium. The program code isexecuted by one or more processors to perform the steps described above.

Various example embodiments have been described above. Alternatives andsubstitutions will now be apparent to persons of skill in the art. Forexample, embodiments using cellular phone applications are described,however, the principles described herein can be applied to other typesof application being executed on mobile devices. Accordingly, it is notintended to limit the invention except as provided by the appendedclaims.

The invention is claimed as follows:
 1. A telecommunications systemcomprising: one or more processors; and a computer readable memorystoring program code which, when executed by the one or more processors,causes the one or more processors to at least: after a communication isestablished by a first electronic communication device and a secondelectronic communication device, receive from the first electroniccommunication device, digital auditory data corresponding to an ongoingconversation between a first person using the first electroniccommunication device and a second person using the second electroniccommunication device, the ongoing conversation including a plurality ofwords spoken by the first person and the second person during theongoing conversation; and while the conversation is ongoing between thefirst person using the first electronic communication device and thesecond person using the second electronic communication device:responsive to content of converted text based on the plurality of wordsspoken: process the converted text to identify and extract an entity tocreate an object defined in an information knowledge model; utilize theobject to enable transformation of the information knowledge model; andutilize the information knowledge model to suggest information for theongoing conversation by: identifying an entity relating to the ongoingconversation, and enabling display of information related to the entityrelating to the ongoing conversation by at least one of the firstelectronic communication device and the second electronic communicationdevice.
 2. The telecommunication system of claim 1, wherein thesuggesting of information for the conversation enables informationrelating to a prior entity to be revisited.
 3. The telecommunicationsystem of claim 1, wherein the displayed information is to clarify ameaning of at least one of: the entity relating to the conversation, arelationship between a plurality of entities, and an entity attribute.